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Abstract  for  Initial  Assessment  of  the  Air  Combat  Environment  Test  and  Evaluation  Facility  To 

Support  Operational  Test  and  Evaluation 

This  paper  briefly  describes  the  Navy's  Operational  Test  and  Evaluation  Force's  (OPTEVFOR's) 
assessment  regarding  the  suitability  of  selected  laboratories  at  the  Air  Combat  Environment  Test  and 
Evaluation  Facility  (ACETEF)  to  support  the  operational  test  and  evaluation  of  electronic  warfare  (EW) 
systems — radar  warning  receivers  (RWRs)  and,  in  particular,  the  Navy's  next-generation  RWR,  the 
AN/ALR-67(V)3. 

The  use  of  hardware-in-the-loop  and  man-in-the-loop  simulation  facilities  is  one  way  to  increase  combat 
realism  and  provide  the  means  to  test  more  fully  EW  and  electronic  combat  capabilities.  Critical 
assumptions  and  model  limitations,  however,  may  adversely  affect  restdts.  Thus,  realistic  testing  in  a 
virtual  world  requires  the  proper  verification  and  validation  (V &V)  of  the  environment  and  simulated 
emitters. 

ACETEF  is  a  full-spectrum  electronic  combat  test  facility.  One  of  ACETEF's  labs — the  Electronic 
Warfare  Integrated  Systems  Test  Lab  (EWISTL)— can  be  linked  with  a  manned  fligfit  simulator  to 
provide  a  fli^t  environment  with  an  emitter  variety  and  emitter  density  that  is  unattainable  on  any  test 
range.  ACETEF  tests  EW  systems  that  are  fully  integrated  aboard  an  operational  mission  aircraft. 

At  the  request  of  the  Commander  OPTEVFOR,  we  have  made  an  initial  assessment  reg^ding  the  suit¬ 
ability  of  selected  laboratories  at  ACETEF  to  support  the  OT&E  of  the  Navy's  next-generation  RWR. 
Our  goal  was  to  determine  whether  ACETEF  provided  a  level  and  quality  of  data  similar  to  those 
obtained  during  operational  testing  at  an  open-air  test  range  and  whether  ACETEF's  test  support 
processes  were  mature  enough  to  ensure  that  the  facility  could  support  the  OT&E  customer. 

Based  on  our  analysis  of  the  models  used,  observations  made  during  the  RWR  testings  and  comparisons 
of  the  ACETEF-derived  data  to  real-world  data,  we  have  identified  and  documented  ACETEF's 
capabilities  and  limitations  to  successfully  support  the  OT&E  of  RWRs.  As  a  result  of  this  assessment, 
OPTEVFOR  is  preparing  a  letter  approving  the  use  of  these  laboratories  at  ACETEF  to  support  the 
OT&E  of  the  Navy's  next-generation  RWR. 

The  assessment  also  highlighted  other  larger  issues  that  must  be  addressed  in  order  to  ensure  the  suc¬ 
cessful  use  of  modeling  and  simulation  resources.  Among  these  larger  issues  are  user  education,  field 
data  to  support  V&V,  data  standardization  needs,  determining  who  should  be  responsible  for 
conducting  (and  funding)  these  overall  assessments  of  models  and  simulations,  extending  the  approach 
to  other  similar  T&E  faculties,  and  correlating  test  results  across  facilities. 

Keywords:  verification  and  validation  (V&V),  modeling  and  simulation,  electronic  warfare,  ACETEF, 
operational  test  and  evaluation,  radar  warning  receiver 
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Introduction 

Many  limitations  associated  with  the  operational  test  and  evaluation  (OT&E)  of  electronic  warfare  (EW) 
systems  make  achieving  the  ^^realistic  combat  conditions'^  demanded  by  law  [1]  either  difficult  or  impos¬ 
sible.  Providing  an  adequate  EW  testing  environment  with  a  sufficiendy  dense  and  varied  emitter 
environment  woidd  involve  large  numbers  of  personnel  and  hardware  and  would  be  cost  prohibitive  in 
nearly  all  instances.  One  way  to  overcome  some  of  the  limitations  is  to  augment  the  available  test 
resources  with  models  and  simulations  (M&S). 

We  view  the  use  of  M&S  to  support  the  OT&E  process  as  a  complement  to  live  testing.  Although  there  is 
no  substitute  for  open-air  fli^t  testings  the  quantity  and  operating  characteristics  of  open-air  range 
emitters  do  not  fully  represent  an  operationally  realistic  EW  environment.  Emitters  may  not  be  available 
on  a  particular  range  (or  on  any  range).  At  an  open-air  range,  civilian  radars  and  other  radio  frequency 
(RF)  signals  are  not  under  test  control.  ^^Live^'  tests,  requiring  large  tracts  of  airspace,  must  be  closely 
coordinated  with  the  Federal  Aviation  Administration  (FAA).  Test  restrictions  must  be  in  place  to  avoid 
inadvertent  civilian  communication  losses  (e.g.,  from  jamming)  or  the  'Hjlindingf'  of  FAA  radars  (e.g., 
from  chaff).  The  controlled  environment  of  a  hardware-in-the-loop  (HITL)  facility  may  help  to  overcome 
these  environmental  and  safety  limitations. 

In  addition,  the  use  of  HTTL  and  man-in-the-loop  (MITL)  simulation  facilities  may  increase  the  combat 
realism  and  provide  the  means  to  test  more  fully  EW  and  electronic  combat  capabilities  of  increasingly 
complex  aircraft  avionics  and  radar  systems.  By  more  fully  testing  these  systems  with  M&S  resources, 
we  are  attempting  to  reduce  further  any  uncertainties  that  the  systems  work  as  they  are  designed.  The 
most  significant  disadvantage  of  these  simulation  resources,  when  compared  to  live  testing,  is  that 
critical  assumptions  and  model  limitations  may  adversely  affect  results.  When  used  as  a  risk  reduction 
measure,  M&S  may  introduce  new  risks.  Thus,  realistic  testing  in  a  virtual  world  demands  that  the 
environment  and  simulated  emitters  be  properly  verified  and  validated  (V &V)* 

At  the  request  of  the  Commander  OPTEVFOR,  we  have  made  an  initial  assessment  regarding  the  suit¬ 
ability  of  selected  components  of  the  Air  Combat  Environment  Test  and  Evaluation  Facility  ( ACETEF)  to 
support  the  OT&E  of  the  Navy's  next-generation  radar  warning  receiver  (RWR),  the  AN/  ALR-67(V)3, 
Our  goal  was  to  determine 

•  Whether  ACETEF  provided  a  level  and  quality  of  data  similar  to  those  obtained  during 
operational  testing  of  an  RWR  at  an  open-air  test  range. 

•  Whether  ACETEF's  test  support  processes  were  mature  enough  to  ensure  that  the  facility 
could  support  the  OT&E  customer. 
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Background 

The  Air  Combat  Environment  Test  and  Evaluation  Facility  (ACETEF)  located  at  the  Naval  Air  Warfare 
Center,  Aircraft  Division  (NAWC-AD),  Patuxent  River,  Maryland,  is  a  full-spectrum  electronic  combat 
test  facility.  ACETEF  tests  EW  systems  that  are  fully  integrated  aboard  an  operational  mission  aircraft. 
ACETEF  consists  of  a  number  of  laboratories  and  a  variety  of  simulations,  each  spedalizmg  in  a 
particular  aspect  of  aircraft  and  emitter  operations.  The  facility  is  capable  of  simulating  a  wide  array  of 
radars,  missiles,  and  jammer  techniques,  all  of  which  may  be  linked  with  a  maimed  flight  simulator  to 
provide  a  flight  environment  with  an  emitter  variety  and  emitter  density  that  is  unattainable  on  any  test 
range. 

Six  components  of  ACETEF  were  essential  to  this  assessment:  the  Electronic  Warfare  Integrated  Systems 
Test  Laboratory  (EWISTL),  the  Operations  and  Control  Center  (OCC),  the  Manned  Flight  Simulator 
(MFS),  the  Simulated  Warfare  Environment  Generator  (SWEG),  tibe  shielded  hangar,  and  the  Aircraft 
Anechoic  Test  Facility  (AATF).  We  did  not  examine  any  of  the  other  laboratories  at  ACETEF  during  this 
assessment. 

EWISTL  simidflips  the  RF  characteristics  of  a  variety  of  emitters.  The  EWISTL  emitter  library  contains 
more  than  5,000  radar  modes.  EWISTL  may  use  six  different  RF  simulator  systems.  For  our  tests,  we 
used  the  Advanced  Tactical  Electronic  Warfare  Environment  Simulator  (ATEWES)  RF  simulator  system. 
ATEWES  is  an  open  loop  system  that  can  generate  more  than  1,000  simultaneous  radar  emitters  by 
multiplexing  15  signal  generators.  The  system  can  generate  up  to  4  million  pulses  per  second.  EWISTL 
can  operate  in  a  scripted-scenaiio  mode,  in  a  dynamic  mode  imder  the  direction  of  SWEG  commands,  or 
in  an  interactive  mode  with  a  console  operator. 

The  CX:C  is  responsible  for  controlling  the  interactions  of  the  various  laboratories,  the  SWEG  simulation, 
and  the  system  under  test.  All  ACETEF  data  collection  is  performed  by  the  OCC.  OCC  personnel 
construct  the  overall  test  scenarios. 

SWEG  is  a  nde-based  model  that  permits  the  modeling  of  players,  platforms,  and  weapon  systems. 
SWEG  controls  the  interactions  among  the  ACETEF  laboratories.  For  example,  SWEG  computes  when 
an  aircraft  is  terrain  masked  from  an  emitter  and  sends  a  control  signal  to  EWISTL  to  inhibit  the 
transmission  of  signal  pulses. 

The  MFS  supports  MTTL  testing.  The  MFS  has  both  desktop  aircraft  simulators  and  state-of-the-art 
large-domed  simulators.  The  domed  simulator  is  coupled  with  a  motion  system  that  displays  computer¬ 
generated  scenes,  providing  the  pilot  with  an  out-of-cockpit  view. 

Aircraft  may  be  tested  in  either  the  shielded  hangar  or  the  AATF.  The  hangar  provides  an  RF-isolated 
environment.  The  anechoic  chamber  provides  an  RF-  and  electromagnetic-free  environment  secure  from 
outside  interference  and  similarly  secure  from  outside  monitoring.  The  chamber  can  safely  absorb  RF 
signals,  which  means  systems  can  be  tested  while  operating  the  aircraft's  radar  and  other  aviomc 
systems. 

ACETEF  has  been  used  extensively  for  developmental  testing  (DT)  of  both  EW  and  avionic  systems, 
including  the  AN/ALR-67(V)3  RWR,  the  Airborne  Self-Protection  Jammer,  and  upgrades  to  the  EA-6B. 
The  EA-6B  program  was  able  to  eliminate  ten  planned  DT  test  flights — saving  milhons  of  dollars  and 
several  weeks  of  effort— through  the  use  of  ACETEF[2]. 


The  AN/ALR-67(V)3  is  the  Navy's  next-generation  RWR.  An  RWR  is  a  passive  detector  designed  to 
intercept,  identify,  and  display  radar  emitter  information  to  the  pilot.  Becatise  the  AN/ ALR-67(V)3  is  in 
early  development  and  no  real-world  performance  data  for  the  RWR  existed  at  the  time  of  this  initial 
assessment,  we  compared  the  ACETEF-generated  residts  to  data  collected  from  the  Navy's  current  fleet¬ 
wide  RWR  system — an  AN/ ALR-67(V)2. 

Approach 

Our  assessment  was  a  formal  examination  of  selected  components  of  ACETEF  to  determine  their 
suitability  to  support  the  OT&E  of  the  Navy's  next-generation  RWR— the  AN/  ALR-67(V)3.  Before  this 
assessment,  ACETEF  had  not  completed  a  formal  documented  V&V  process.  Previous  informal  V&V 
efforts  of  the  facility  had  largely  been  undocumented. 

We  structured  and  executed  a  series  of  tests  using  ACETEF  and  an  F/A-18  equipped  with  the 
AN/ALR-67(V)2.  We  then  compared  the  ACETEF-generated  results  to  real-world  data  collected 
previously  from  an  F/A-18  equipped  with  an  AN/ALR-67(V)2.  We  mtist  note  that  we  had  a  limited 
amount  of  real-world  performance  data  and  that  the  AN/ALR-67(V)2  is  an  imperfect  "yardstick"  for 
detecting,  identifying^  and  displaying  emitter  information.  For  the  assessment,  the  emitters  of  the 
Electronic  Combat  Range  (ECR),  located  at  the  Naval  Air  Warfare  Center,  Weapons  Division,  (NAWC- 
WD)  China  Lake,  California,  were  simulated.  The  real-world  test  flights  were  flown  at  ECR.  We 
conducted  the  ACETEF  tests  in  three  phases.  Each  of  the  phases  used  the  CX^C  to  collect  data  from  both 
the  simulations  and  the  RWR  aboard  the  aircraft. 

The  first  phase  consisted  of  two  sets  of  tests  using  only  EWISTL  to  generate  various  emitter  modes. 
Although  this  phase  was  artificial  by  OT&E  standards,  these  tests  provided  us  an  efficient  way  to 
simulate  a  variety  of  emitter  modes  and  to  collect  sufficient  data  to  establish  the  basic  simulation 
performance  of  the  ACETEF.  We  attempted  to  maximize  the  number  of  emitters  and  their  operating 
modes,  while  limiting  the  amount  of  computer  time  required  to  complete  the  tests  and  minimizing  the 
amount  of  analytic  effort  required  to  process  the  data. 

The  second  phase  consisted  of  replicating  seven  test  flights  flown  earlier  at  ECR.  In  this  phase,  we  used 
both  EWISTL  to  generate  the  emitter  modes  and  SWEG  to  model  the  aircraft's  flightpath  and  to  control 
the  EWISTL-generated  radars. 

The  third  phase  tested  ACETEF's  ability  to  simulate  a  dynamic  EW  environment— typical  of  those 
possible  in  open-air  flight  tests  at  ECR.  F/A-18  pilots  from  VX-9,  China  Lake,  using  the  MFS  "flew" 
through  a  virtual  ECR  range  in  eight  basic  scenarios.  SWEG  controlled  all  of  the  interactions  among  the 
virtual  airspace,  the  MFS,  and  the  EWISTL-generated  radars. 

In  addition  to  the  operational  performance  data  derived  from  the  three-phase  test,  we  also  evaluated 
ACETEF's  ability  to  support  the  OT&E  customer  by  examining  its  day-to-day  procedures,  testing 
processes,  and  facility  documentation. 

During  the  assessment,  OPTEVFOR  and  ACETEF  personnel  carried  on  a  continual  dialogue. 
Discrepancies,  as  they  were  noted,  were  referred  back  to  ACETEF.  At  the  end  of  the  assessment,  we 
made  recommendations  to  the  ACETEF  personnel  regarding  improvements  in  the  facility  to  better 
support  OT&E  needs. 


Results 


We  found  that,  in  general,  ACETEF  can  generate  signals  of  the  right  amplitude  and  fidelity  for  accurate 
detection,  identification,  and  display  by  the  RWR  as  compared  to  real-world  performance.  We  based  our 
condusion  about  ACETEF  on  the  same  criteria  used  in  open-air  flight  tests  (e.g.,  what  was  the  range 
when  an  emitter  was  detected  and  what  percentage  of  the  emitters  were  correctly  identified  by  the 
RWR).  In  general,  the  ACETEF-derived  results  were  comparable  to  open-air  flight  test  results.  In  a 
limited  number  of  cases,  we  foimd  differences  between  the  ACETEF-derived  data  and  real-world 
AN/ALR-67(V)2  data.  These  differences  were  due  to  slight  inaccurades  in  certain  radar  modes.  After 
our  assessment,  EWISTL  personnel  made  the  necessary  changes  to  these  modes,  although  their 
modificatioiis  have  yet  to  be  verified  and  validated. 

Overall,  the  variety  of  emitter  modes  and  the  technical  accuracy  of  the  emitter  parameters  were 
satisfactory  to  support  the  needs  of  OT&E. 

In  examining  ACETEF's  ability  to  replicate  selected  flight  segments  of  actual  flight  tests  at  ECR,  our 
analysis  indicated  dose  agreement  with  flight  test  events  and  sequences.  We  judged  ACETEF  to  be 
satisfactory  in  generating  these  kinds  of  scripted  scenarios  to  support  OT&E  requirements. 

We  initially  assessed  ACETEF's  related  documentation  as  unsatisfactory  and  made  a  number  of 
recommendations  regarding  desired  documentation.  Since  the  beginning  of  the  assessment,  ACETEF 
personnel  have  taken  aggressive  action  to  improve  their  documentation.  These  steps  indude 

•  Starting  an  independent  V&V  of  the  EWISTL  facility. 

•  Developing  a  formal  configuration  management  plan. 

•  Developing  user's  guides  for  each  laboratory.  These  guides,  currendy  in  draft  form,  will 
assist  future  users  by  explaining  the  functionality  of  each  laboratory  and  discussing  known 
limitations — induding  any  impacts  these  limitations  may  have  on  potential  user  test 
objectives. 

We  also  noted  limitations  with  the  MFS.  The  aircraft  model  was  limited  to  a  small  set  of  external 
weapon  stores/drag  coeffidents.  In  addition,  the  drag  coeffidents  were  not  dynamic  (e.g.,  dropping 
external  fuel  tanks  did  not  reduce  drag).  We  viewed  this  as  a  minor  limitation.  Furthermore,  the  aircraft 
engine  model  did  not  accelerate  and  decelerate  correctly.  Also  the  pilot  can  exceed  real-world 
performance  by  exceeding  normal  engine  throttle  settings.  Again,  since  our  assessment,  the  engine 
model  has  been  improved,  but  its  performance  has  yet  to  be  formally  validated. 

We  documented  our  efforts  during  the  assessment  in  considerable  detail  so  that  future  users  may 
leverage  from  this  work.  We  hope  that  future  V&V  efforts  of  ACETEF  are  also  weD-documented.  The 
operational  tester  requires  very  detailed  information  to  either  successfully  simulate  an  event  or  avoid  a 
potential  pitfall.  For  example,  at  some  frequencies  the  ATEWES  signal  generators  may  be  unable  to 
generate  sufficient  power  to  replicate  fully  the  power  received  by  the  RWR  when  the  aircraft  gets  dose 
to  an  emitter.  Figure  1  shows  the  minimum  dosing  distance  between  an  aircraft  and  an  emitter  that 
ATEWES  can  successfully  simulate  across  a  given  frequency  spectrum.  The  graph  is  construded  from 
knowing  the  effective  radiated  power  (ERP)  of  the  emitter  being  simulated,  the  maximum  power  output 
of  the  ATEWES  signal  generators  for  a  set  of  frequendes,  and  the  one-way  radar  range  equation.  When 
an  aircraft  crosses  below  the  graph,  ATEWES  wiU  be  unable  to  generate  suffident  power  to  successfully 
model  the  emitter.  When  the  aircraft  "flies"  within  this  minimum  dosing  distance,  the  power  received 


by  the  RWR  will  remain  constant.  This  lack  of  increase  in  the  received  power  may  adversely  affect  the 
test  restdts. 

From  figure  1,  if  the  tester  wants  to  simulate  an  emitter  whose  operating  frequencies  range  between  200 
and  300  frequency  units  and  the  tester  also  wants  to  simulate  flying  the  aircraft  between  10  and  15 
distance  units  from  the  emitter,  then  the  tester  needs  to  avoid  those  frequencies  between  250  to  265.  In 
this  example,  at  a  firequency  of  approximately  260,  the  RWR  will  not  receive  any  additional  signal  power 
when  it  flies  within  30  distance  units  of  the  simulated  radar.  Without  detailed  knowledge  of  the 
capabilities  and  limitations  of  the  simulation,  the  operational  tester  may  be  unable  to  choose  the 
appropriate  fi*equencies  for  a  successful  test  simulation. 


0  100  200  300  400 

Frequency 


Figure  1.  Closest  distance  between  emitter  and  aircraft  that  ATEWES  signal  generator  can  successfully 
simulate 

We  found  that  the  quantity  and  type  of  ACETEF-derived  RWR  data  are  comparable  to  the  data  obtained 
from  an  instrumented  aircraft  flown  at  ECR.  The  ACETEF-collected  emitter  data  have  greater  resolution 
and  precision  than  emitter  data  collected  at  an  open-air  test  range,  particularly  for  noninstrumented 
emitters  at  a  test  range. 

During  our  assessment,  we  found  that  the  procedures  in  place  at  ACETEF,  although  they  could  be 
improved,  were  good  overall.  The  facility  and  the  personnel  are  well-suited  to  support  the  OT&E 
customer.  ACETEF  personnel  have  been  very  receptive  to  our  recommendations  and  the  needs  of  the 
OT&E  community.  As  a  result,  ACETEF  personnel  have  begun  to  make  improvements  in  their  test 
conduct,  emitter  database  management,  and  model  algorithms — in  addition  to  the  previously 
mentioned  actions  that  they  are  undertaking. 

Based  on  our  analysis  of  the  models  used,  observations  made  during  the  RWR  testing,  and  comparisons 
of  the  ACETEF-derived  data  to  real-world  data,  we  have  identified  and  documented  ACETEF's 
capabilities  and  limitations  to  successfully  support  the  OT&E  of  RWRs.  As  a  result  of  this  assessment, 
OPTEVFOR  is  preparing  a  letter  approving  the  use  of  these  laboratories  at  ACETEF  to  support  the 
OT&E  of  the  AN/ ALR-67(V)3. 


Advantages  of  using  ACETEF 

ACETEF  is  not  a  substitute  for,  but  rather  a  complement  to  live  testing  at  an  open-air  range.  "Together, 
models  and  simulations  offer  the  ability  to  expand  the  operational  assessment,  while  field  testing  offers 
the  means  to  more  effectively  calibrate  and  validate  the  models"  [3].  ACETEF's  laboratories  provide  the 
following  benefits: 

•  Provide  for  adequate  preparation  of  open-air  test  flights,  helping  to  ensure  that  a  live  test  is 
done  correctly  the  first  time  and  that  the  desired  data  can  be  collected. 

•  Allow  the  tester  to  tailor  the  open-air  tests  for  resolving  questions  that  cannot  be 
satisfactorily  answered  through  simulation.  Significant  costs  may  be  avoided,  if  unnecessary 
marginal  tests  can  be  identified  through  the  analysis  of  simulation-derived  results. 

•  Provide  realistic  testing  earlier  in  the  life  cyde,  thus  allowing  more  timely  discovery  and 
resolution  of  errors.  During  the  quick-look  operational  assessment  of  the  AN/ALR-67(V)3 
performed  at  ACETEF,  it  was  quickly  determined  that  the  RWR  was  not  ready  for  open-air 
test  flights  at  that  time.  The  decision  was  made  to  postpone  the  test  flights  and  the  program 
office  avoided  wasting  nearly  $200,000  in  ECR  range  and  associated  flight  costs  for  a 
premature  test.  ACETEF  costs  were  about  $20,000. 

•  Allows  the  tester  to  investigate  problems  arising  in  flight  tests  by  replicating  the  tests 
through  simulation. 

•  Provide  a  means  for  generating  suffident  data,  possibly  interpolating  available  real-world 
data  to  fiU  open-air  test  flight  gaps. 

•  Provide  increased  security  by  operating  in  a  dosed  facility,  which  permits  the  testing  of 
classified  equipment  without  compromise. 

•  Provide  unique  testing  scenarios  in  a  sdentifically  controlled  envirorunent. 

•  Provide  emitters  both  in  large  numbers  and  in  a  variety  of  types. 

■  •  Extend  the  testing  environment  by  sunulating  more  realistic  scenarios,  involving  more 
advanced  radars  in  a  higher  pulse  density  environment,  than  can  be  provided  by  a  test 
range.  This  may  be  especially  true  when  simulating  complex  combat  environments 
incorporating  '^future^^  emitters. 

These  ACETEF  laboratories  now  become  one  more  set  of  tools  in  the  operational  tester's  toolbox.  The 
use  of  these  laboratories  should  translate  into  a  more  cost-effective  means  to  deliver  a  more  thorougjhly 
tested  system  to  the  fleet.  In  addition,  by  extending  the  test  environment,  fewer  limitations  to  the  scope 
of  testing  should  be  required  (albeit  the  tester  must  be  cognizant  of  the  fadlit/ s  limitations). 

Larger  issues 

The  assessment  also  highlighted  other  larger  issues  that  must  be  addressed  to  ensure  the  successful  use 
of  M&S  resources. 

The  EW  test  facilities  should  adopt  common  standards  which  include  emitter  naming  conventions  and 
common  formats  for  data  collection.  Common  standards  will  facilitate  the  analysis  of  the  data— using 
common  analytic  tools— and  understanding  of  the  results,  regardless  of  whether  the  data  are  obtained  in 


a  laboratory  or  at  an  open-air  test  range.  A  uniform  data  format  could  enhance  the  correlation  of  test 
restdts  across  facilities. 

There  is  a  need  for  a  user  education  process.  End-users,  such  as  an  operational  test  director  at  a  test 
squadron  like  VX-9,  must  be  made  aware  of  what  M&S  capabilities  and  tools  are  available  and  how 
these  simulations  have  been  used — both  successfully  and  unsuccessfully — in  the  past.  Furthermore, 
users  must  be  provided  detailed  documentation  about  the  limitations  and  assumptions  that  each 
simulation  brings  with  it,  and  must  understand  how  these  limitations  and  assumptions  can  affect  the 
outcome  of  a  test. 

This  assessment  highlights  the  question  of  who  should  be  responsible  for  overall  assessments — in  terms 
of  both  performing  the  assessment  and  funding  it.  OPTEVFOR  performed  the  assessment,  with  funding 
assistance  from  both  ACETEF  and  the  AN/  ALR-67(V)3  program  office,  PMA-272.  However,  is  this  type 
of  assessment  something  for  which  the  end-user  is  responsible,  an  activity  for  which  the  individual 
facility  should  be  idtimately  responsible,  or  wotdd  an  organization  specializing  in  V&V  efforts  be  a 
better  choice?  In  general,  end-users  may  have  neither  the  desire  nor  the  resomces  to  perform  formal 
V&V  of  models.  A  facility  may  have  exceptional  technical/ software  expertise,  but  may  not  have  the 
personnel  and  expertise  for  V&V  efforts.  A  V&V  organization  may  be  well- versed  in  documentation 
standards  and  V&V  techniques,  but  may  lack  the  technical  expertise  to  fully  capture  the  critical 
limitations  of  the  model. 

We  view  the  V&V  process  as  reducing  the  risk  associated  with  using  a  particular  model.  Although  total 
confidence  in  the  model  is  a  desirable  goal,  for  a  complex  simulation  this  goal  may  be  prohibitively 
expensive.  The  modeling  community  needs  to  recognize  that  V&V  is  not  an  all-or-nothing  proposition, 
but  rather  is  a  dynamic  process.  Each  program  will  dictate  the  amount  of  V&V  effort  required.  Our 
assessment  of  ACETEF's  laboratories  lays  along  the  V&V  continuum.  We  have  documented  our  efforts 
during  this  limited  assessment  in  considerable  detail  so  that  future  users  may  build  on  this  work.  We 
must  view  these  documents  as  living  documents,  reflecting  a  continuing  and  refining  V&V  effort. 

The  V&V  problem  is  compounded  because  the  M&S  community  lacks  supporting  test  data.  Obtaining 
sufficient  usable  real-world  data  was  critical  in  preparing  for  a  validation  of  the  laboratories  involved  in 
the  assessment.  Although  we  enjoyed  excellent  cooperation  from  ECR  and  VX-9  personnel  in  obtaining 
real-world  AN/ALR-67(V)2  data,  it  was  still  an  arduous  task.  A  repository  of  previously  collected  data 
would  have  been  an  invaluable  resource.  In  addition,  we  note  the  lack  of  a  conduit  to  feedback  real- 
world  data  to  the  modelers.  In  the  model-test-model  paradigm,  data  collected  during  live  tests  should 
be  used  to  refine  and  validate  the  associated  environmental,  threat,  and  system  models.  These  data  are 
particularly  important  for  developing  models  of  "leading  edge"  systems,  with  which  the  T&E 
community  primarily  deals.  Although  prior  experience  with  a  similar  system  under  test  can  help  to 
provide  some  validation  of  a  model's  basic  algorithms,  the  precise  performance  of  the  system  is  an 
educated  guess  until  real-world  data  become  available  through  prototypes  and  testing.  As  more  data 
become  available,  the  level  of  V&V  can  be  raised.  Performance  data  from  the  AN/ ALR-67(V)3  should 
begin  to  be  archived  for  future  V&V  efforts  involving  this  RWR. 

We  believe  that  the  open  cooperation  among  ACETEF,  ECR,  OPTEVFOR,  and  the  assessors  should 
serve  as  an  example  for  future  assessment  efforts  of  complex  M&S  resources.  In  addition,  we  believe 
that  this  approach  shotdd  be  applied  to  other  similar  M&S  resources  used  by  T&E.  V&V  efforts  will  help 
to  create  a  standardization  among  EW  simulation  facilities  that  reflects  real-world  performance  data. 
This  standardization  will  help  to  ensure  a  correlation  of  results  across  T&E  facilities.  Users  should  be 
able  to  depend  on  some  level  of  consistency  among  modeb  with  the  "same"  functionality— whether  the 
data  are  derived  from  a  digital  model,  an  HH  L/MITL  facility  like  ACETEF,  or  from  open-air  test  flights. 


Summary 

This  paper  briefly  describes  OPTEVFOR's  assessment  regarding  the  suitability  of  selected  components 
of  ACETEF  to  support  the  operational  test  and  evaluation  of  EW  systems— RWRs  and,  in  particular,  the 
Navy's  next-generation  RWR,  the  AN/ALR-67(V)3.  As  a  result  of  this  assessment,  OPTEVFOR  is 
preparing  a  letter  approving  the  use  of  the  laboratories  at  ACETEF,  examined  during  this  assessment,  to 
support  the  OT&E  of  the  AN/ALR-67(V)3.  We  believe  this  assessment  is  an  important  first  step  in 
formally  documenting  the  capabilities  and  limitations  of  ACETEF.  We  have  extensively  documented  our 
findings  in  order  that  others  might  build  on  and  extend  these  restdts. 

Both  OPTEVFOR  and  ACETEF  have  chosen  to  invest  part  of  their  resources  in  a  project  that  will 
provide  long-term  benefits  to  the  Navy  and  the  Department  of  Defense  as  a  whole.  We  believe  it  is  only 
this  type  of  long-term  vision  and  investment  in  the  necessary  infi-astructure  that  will  permit  M&S  to  be 
used  most  efficiently  and  most  effectively. 

It  is  less  important  that  we  found  limitations,  but  rather  that  the  facility's  capabilities  and  limitations  are 
now  documented  and  better  understood.  This  information  is  invaluable  to  the  tester  who  desires  to  fiiUy 
exploit  the  facility's  strengths,  while  attempting  to  work  around  its  weaknesses.  Similarly,  our  findings 
are  helping  guide  the  ACETEF  personnel  in  improving  their  facihty. 
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suits  of  Our  Analysis 
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Summary  of  Our  Analysis 
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Advantages  of  Using  ACETEF 
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•  Identify  early  /  reduce  extent  ot 
scope  of  testing 

•  More  thoroughly  tested  system 
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Can  ACETEF  Support  Operational 
Test  and  Evaluation? 
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Larger  issues 
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Points  of  Contact 
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